An Ensemble Approach to Combining Expert Opinions

This paper introduces a new classification problem in the context of human computation. Given training data annotated by m human experts s.t. for each training instance the true class is provided, the task is to estimate the true class of a new test instance. To solve the problem we propose to apply a wellknown ensemble approach, namely the stacked-generalization approach. The key idea is to view each human expert as a base classifier and to learn a meta classifier that combines the votes of the experts into a final vote. We experimented with the stacked-generalization approach on a classification problem that involved 12 human experts. The experiments showed that the approach can outperform significantly the best expert and the majority vote of the experts in terms of classification accuracy.